WORLD INTELLECTUAL PROPERTY ORGANIZATION 
Internationa] Bureau 




PCT 

INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification 6 : 
G09B 19/16, 9/08, 9/24 



Al 



(11) International Publication Number: WO 97/34277 

(43) International Publication Date: 18 September 1997 (18.09.97) 



(21) International Application Number: PCT/US97/03560 

(22) International Filing Date: 7 March 1997 (07.03.97) 



(30) Priority Data: 

08/614,337 



12 March 1996(12.03.96) 



US 



(60) Parent Application or Grant 
(63) Related by Continuation 
US 

Filed on 



08/614,337 (CIP) 
12 March 1996(12.03.96) 



(71) Applicant (for all designated States except US): TRAIN- 

ING INNOVATIONS GROUP, LL.C. [US/US]; P.O. Box 
19833, Charlotte, NC 28219 (US). 

(72) Inventor; and 

(75) Inventor/Applicant (for US only): NEMETH, Louis, G. 
[US/US]; 57 Heritage Drive, Lake Wylie, SC 29710 (US). 

(74) Agents: TAULBEE, George, M. et al.; Bell, Seltzer, Park & 
Gibson, P.O. Drawer 34009, Charlotte, NC 28234 (US). 



(81) Designated States: AL, AM, AT, AT (Utility model), AU, AZ, 
BA, BB, BG, BR, BY, CA, CH, CN, CU, CZ, CZ (Utility 
model), DE, DE (Utility model), DK, DK (Utility model), 
EE, EE (Utility model), ES, FI, FI (Utility model), GB, GE, 
HU, IL, IS, JP, KE, KG, KP, KR, KZ, LC, LK, LR, LS, 
LT, LU. LV, MD, MG, MK, MN, MW, MX, NO, NZ, PL, 
PT, RO, RU, SD, SE, SG, SI, SK, SK (Utility model), TJ, 
TM, TR, TT, UA, UG, US, UZ, VN, ARIPO patent (GH, 
KE, LS, MW, SD, SZ, UG), Eurasian patent (AM, AZ, BY, 
KG, KZ, MD, RU, TJ, TM), European patent (AT, BE, CH, 
DE, DK, ES, FI, FR, GB, GR, IE, IT, LU, MC t NL, PT, 
SE), OAPI patent (BF, BJ, CF, CG, CI, CM, GA, GN, ML, 
MR, NE, SN, TD, TG). 



Published 

With international search report. 
Before the expiration of the time limit for amending the 
claims and to be republished in the event of the receipt of 



(54) Title: DEBRIEFING SYSTEMS AND METHODS FOR RETRIEVING AND PRESENTING MULTIPLE DATASTREAMS WITH 
TIME INDICATION MARKS IN TIME SYNCHRONISM 



60 



VIDEO \U 
& AUDIO 
RECORD \J 



50%JMARKING 
SYSTEM 



/22 
MIC 



10 



CHECK PILOT 
INTERFACE 
WITH HOST 
COMPUTER 

v 40 




(57) Abstract 

Event debriefing systems and methods are provided for recording a plurality of datastreams related to an event Each datastream 
includes chronological information related to at least one aspect of the event An observer of the event may create time indication marks 
that facilitate recalling, during debriefing, what transpired al any particular point in time during the event The recorded datastreams may 
be retrieved and presented beginning at marked time indications and continued chronologically thereafter in time synchronism. 



1 



FOR THE PURPOSES OF INFORMATION ONLY 



Codes used to identify States party to the PCT on the front pages of pamphlets publishing international 
applications under the PCT. 



AM 


Armenia 


GB 


United Kingdom 


MW 


Malawi 


AT 


Austria 


GE 


Georgia 


MX 


Mexico 


AU 


Australia 


GN 


Guinea 


NE 


Niger 


BB 


Barbados 


GR 


Greece 


NL 


Netherlands 


BE 


Belgium 


HU 


Hungary 


NO 


Norway 


BF 


B uric ins Paso 


IE 


Ireland 


NZ 


New Zealand 


BG 


Bulgaria 


IT 


Italy 


PL 


Poland 


BJ 


Benin 


JP 


Japan 


FT 


Portugal 


BR 


Brazil 


KE 


Kenya 


RO 


Romania 

Russian Federation 


BY 


Belarus 


KG 


Kyrgystan 


RU 


CA 


Canada 


KP 


Democratic People's Republic 


SD 


Sudan 


CF 


Central African Republic 




of Korea 


SE 
SG 


Sweden 


CG 


Congo 


KR 


Republic of Korea 


Singapore 


CH 


Switzerland 


KZ 


Kazakhstan 


SI 


Slovenia 


CI 


COte d* J voire 


U 


Liechtenstein 


SK 


Slovakia 


CM 


Cameroon 


LK 


Sri Lanka 


SN 


Senegal 


CN 


China 


LR 


Liberia 


sz 


Swaziland 


CS 


Czechoslovakia 


LT 


Lithuania 


TO 


Chad 


CZ 


Czech Republic 


LU 


Luxembourg 


TG 


Togo 


DE 


Genu any 
Denmark 


LV 


Latvia 


TJ 


Tajikistan 


DEC 


MC 


Monaco 


TT 


Trinidad and Tobago 


EE 
ES 


Estonia 


MD 


Republic of Moldovi 


UA 


Ukraine 


Spain 


MG 


Madagascar 


VG 


Uganda 


n 


Finland 


ML 


Mali 


CS 


United States of America 


FR 


France 


MN 


Mongolia 


UZ 


Uzbekistan 


GA 


Gabon 


MR 


Mauritania 


VN 


Viet Nam 



WO 97/34277 



PCT/US97/03560 



-1- 



DEBRIEFING SYSTEMS AND METHODS FOR RETRIEVING 
AND PRESENTING MULTIPLE DATASTREAMS WITH TIME 
INDICATION MARKS IN TIME SYNCHRONISM 

Field of the Invention 

The present invention relates generally to 
event recording systems, and more particularly to 
systems for debriefing event participants. 

5 Background of the Invention 

It is well known throughout the airline 
industry that flight crew error is a contributing 
factor in a large number of aircraft accidents. 
However, many of the errors made by mult i -person flight 
10 crews have very little to do with the technical aspects 
of flying an aircraft. Instead, a large number of 
these errors are related to poor group decision making, 
ineffective communication, inadequate leadership, and 
poor task management (collectively referred to as 

15 "human factors" errors) among the flight crew. 

Flight simulators were developed in the late 
1960's and early 1970' s to increase flight crew 
proficiency in the technical aspects of flying. Flight 
simulators are computer controlled devices which 

20 simulate actual flying conditions of an aircraft and 

train flight crews to deal with a variety of situations 
before they actually encounter them in a real aircraft. 
Early on, the airline industry recognized the 
importance of increasing flight crew proficiency with 

25 respect to eliminating human factors errors, and began 
combining both technical and human factors training 
during flight simulator exercises. For example, a 
high- stress environment that requires clear, 
unequivocal communication between all of the crew 

30 members is the loss of an engine on take off. Although 



WO 97/34277 



PCT/US97/03560 



-2- 

often regarded as a technical maneuver, it also 
requires accurate information transfer between the 
flight crew, the control tower, and the cabin crew. 

Increasing the proficiency of flight crews 
5 with respect to human factors knowledge and skills has 
come to be known as "cockpit resource management" (CRM) 
training. CRM training typically involves videotaping 
a flight simulator training exercise for later playback 
and review by the. flight crew during debriefing. 

10 Simulators equipped with video cameras can help flight 
crews sharpen their communication skills and other 
human factors skills in various emergency situations. 
Unfortunately, conventional video cassette recorder 
(VCR) technology presently used to record flight 

15 simulator exercises is often inadequate during 

debriefing. Rewinding and forwarding is often required 
in order to locate a particular portion of an event 
recorded on a videotape, and to review a portion of the 
videotape multiple times. Some degree of time lag is 

2 0 typically involved because it is rather difficult to 
quickly jump back and forth between different portions 
of the videotape. Another disadvantage of using 
conventional VCR technology for capturing a flight 
simulator training exercise for debriefing purposes is 

25 that it is somewhat difficult to simultaneously capture 
with video cameras, flight data as represented by the 
various indicators and dials in the simulator, and the 
actions of the flight crew. 

Summary of the Invention 

30 It is therefore an object of the present 

invention to provide improved debriefing systems and 
methods . 

It is another object of the present invention 
to provide improved debriefing systems and methods that 
35 overcome the limitations and drawbacks of conventional 
VCR technology. 
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These and other objects are accomplished, 
according to the present invention, by event debriefing 
systems and methods for recording and replaying various 
types of information generated during an event, both 
5 simulated and real. Facilities are provided for 
recording a plurality of datastreams related to an 
event, each datastream including chronological 
information related to at least one aspect of the 
event . Recorded datastreams include video recordings 

10 of the actions of event participants, and audio 
recordings of audible communications of event 
participants. Also included are instrumentation 
datastreams from various event-monitoring instruments, 
including those necessary to create aircraft flight 

15 animation. 

Facilities are provided whereupon, any point 
in time during an event can be "indicated" with a time 
indication mark, created either by an observer or 
automatically by the debriefing system. An observer of 
20 the event may create time indication marks that 
facilitate recalling, during debriefing, what 
transpired at any particular point in time during the 
event. The observer is presented with, preferably via 
a computer display, menus of categories and sub- 
25 categories of pre-defined time indication marks that 
can be selected and associated with the event. 
Similarly, the debriefing system may be configured to 
automatically create time indication marks when certain 
parameters are met during the event. 
30 Time indication marks may be defined for 

virtually any aspect of an event including, but not 
limited to, categories related to the actions of 
participants in the event (CRM for example) , and 
categories related to technical aspects of the event. 
35 By associating a time during the event with a 

particular category of time indication marks, selective 
replay of the various recorded datastreams is 
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facilitated during debriefing. For example, if 
communications between event participants is good (or 
bad) at a particular point in time during an event, a 
pre-defined time indication mark may be selected from a 
5 menu and associated with each of the recorded 

datastreams. During debriefing, the event participants 
can view the datastreams in time synchronism beginning 
at or near this time indication mark. Time indication 
marks that are not pre-defined may also be created and 

10 associated with the recorded datastreams. For example, 
if something were to occur during an event that would 
be worth reviewing during debriefing, yet does not fall 
within a particular pre-defined category, an observer 
can create a time indication mark anyway. 

15 The recorded datastreams are retrieved and 

presented in time synchronism. In particular, the 
recorded datastreams can be retrieved and presented 
beginning at marked time indications and continued 
chronologically thereafter in time synchronism. During 

20 debriefing, a user is presented with various computer 
screens from which particular time indication marks can 
be selected. For example, a list of all marks created 
during an event may be listed. Alternatively, various 
' data from an event, with associated time indication 

25 marks, may be presented for selection to a user. By 
selecting a particular time indication mark, the 
various recorded datastreams are replayed in time 
synchronism beginning at or near the time associated 
with the selected time indication mark. 

30 A display having a plurality of zones is used 

to present video and instrumentation datastreams in 
time synchronism. Event -monitoring instrumentation and 
other data is displayed in dynamic animation. An audio 
datastream is performed in time synchronism with the 

35 displayed video and instrumentation datastreams. 

Consequently, the disadvantages of conventional VCR 
technology are overcome by the present invention. Any 
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portion of a recorded event can be reviewed at the 
press of a button, with virtually no time lag. 

The present invention may be used in 
conjunction with aircraft flight simulator debriefing 
5 systems and methods. Data, from both the flight 

simulator and the video camera on the simulator flight 
deck, is directed to a computer. A computer screen is 
configured to display a variety of pre-defined 
categories of time indication marks. A training 
10 facilitator, referred to as a "check pilot", observes 
crew interactions during the simulator exercise and 
selects time indication marks from the display. The 
time indication marks may be associated with various 
aspects of the simulated flight, including technical 
15 and CRM aspects. For example, if the check pilot 

observes flight crew behavior during a simulated event 
that would be desirable to review during debriefing, he 
or she can select a pre-defined time indication mark, 
or create a mark that is not pre-defined, and associate 
20 the mark with the recorded datastreams. 

Software associated with the debriefing 
computer automatically reconstructs the simulator 
flight, links it to the video and audio signal, from the 
flight deck and displays the digitized video, dynamic 
25 animation and selected instrumentation on the monitor 
located in a debriefing room. During debriefing 
following a simulator exercise, a check pilot is able 
to go to any event during the flight and show the crew 
exactly what transpired on the flight deck, complete 
30 with animated instrumentation and multiple views of the 
aircraft and flight path, all on a single monitor. 
Voice recognition software may also be included wherein 
the conversations of the crew can be searched for 
selected words and phrases during the simulator 
35 exercise. 

The present invention is advantageous because 
it substantially increases training efficiency and 
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effectiveness by allowing the crew to "relive" their 
flight experience in a single integrated display of 
their technical and human factor performance. Another 
advantage of the debriefing system, according to the 
5 present invention, is that a flight crew member can 
debrief himself or herself after a training exercise. 
This removes the subjectivity of the check pilot and 
puts him or her in the position of having to simply 
manage the interaction between the flight crew and the 

10 simulator system. As an additional benefit, the 
present invention will result in increased 
standardization of check pilot briefings, debrief ings, 
and evaluations. 

Prior to the present invention, marks on 

15 videotape lacked identifying information. The check 
pilot knew that a mark existed, but there was not any 
descriptive information associated with the mark. Time 
indication marks, according to the present invention, 
facilitate the recall of events that transpired in the 

20 simulated exercise, particularly at critical junctures, 
when the exercise is replayed during the debriefing 
session. Because there is a tremendous amount of 
stimuli occurring (for example, the flight instruments 
move dynamically, the visual scenes outside of the 

25 cockpit move, and various audio communications take 
place, including bells, horns, communications between 
the flight crew and with air traffic control) during a 
simulated flight exercise, it is difficult for the 
flight crew and the check pilot to recall all of the 

30 events and data that occurred during a session. The 
cognitive processes of the flight crew and the check 
pilot cannot assimilate all the stimuli in order to 
effectively train motor skills and cognitive skills. 

As described above, the invention implements 

35 systems and methods for: (1) recording, marking, and 
presenting multiple datastreams in time synchronism; 
(2) recording, categorized marking, and presenting 
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multiple datastreams in time synchronism; and (3) 
recording, automatically marking, and presenting 
multiple datastreams in time synchronism. Each of 
these aspects may be used individually or in 
5 combination, to improve training efficiency and 
effectiveness during debriefing of an event. 

The present invention is not limited to the 
recording and debriefing of simulated events. Many 
"real" events can be recorded and the actual event 

10 participants debriefed using the present invention. 
For example, the systems and methods of the present 
invention can be utilized in the control room of a 
nuclear power plant, the operating room of a hospital, 
the control center of the National Aeronautics and 

15 Space Administration (NASA) during a space mission, the 
assembly line of a manufacturing plant, and many other 
locations involving both technical aspects of an event 
and interaction among participants in the event. The 
present invention is a powerful learning tool with many 

20 potential applications. 

Brief Description of the Drawings 

Pig. 1A schematically illustrates operations 
for recording, marking, and retrieving multiple 
datastreams, according to the present invention. 
25 Fig. IB schematically illustrates operations 

for marking an indication of time during an event, 
according to the present invention. 

Pig. 1C schematically illustrates operations 
for retrieving and presenting multiple datastreams in 
3 0 time synchronism via time indication marks, according 
to the present invention. 

Fig. 2 schematically illustrates a hardware 
configuration of the present invention when used in 
conjunction with a flight simulator. 
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Fig. 3 is a hierarchical menu of time 
indication marks when the present invention is used in 
conjunction with a flight simulator. 

Pig. 4 illustrates an integrated debriefing 
5 computer display according to the present invention, 
when used in conjunction with a flight simulator. 

Pig. 5 illustrates a display screen for 
selecting a particular time mark from a list of time 
indication marks during debriefing. 
10 Fig. 6 illustrates an alternative display 

screen for selecting time indication marks during 
debriefing. 

Fig. 7 is a hierarchical menu of time 
indication marks according to another aspect of the 
15 present invention. 

Detailed Description of Preferred Embodiments 

The present invention now is described more 
fully hereinafter with reference to the accompanying 
drawings, in which preferred embodiments of the 

2 0 invention are shown. This invention may, however, be 
embodied in many different forms and should not be 
construed as limited to the embodiments set forth 
herein; rather, these embodiments are provided so that 
this disclosure will be thorough and complete, and will 

25 fully convey the scope of the invention to those 
skilled in the art. 

Referring now to Figs. 1A, debriefing 
operations of the present invention include recording 
multiple datastreams related to an event (Block 100) , 

30 marking an indication of time during the event (Block 
200) , and retrieving and presenting the recorded 
datastreams in time synchronism instantly via the 
marked indication of time (Block 300) . The multiple 
datastreams may include video, audio, event -monitoring, 

35 and instrumentation datastreams. The debriefing system 
has the capability of digitizing and synchronizing 
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multiple datastreams, displaying the data 
simultaneously on a computer screen, and permitting 
instantaneous access to information within each 
datastream for any particular point in time. 
5 Referring now to Pig. IB, operations for 

marking an indication of time during an event will now 
be described. An observer of an event can associate a 
time indication mark with recorded information to 
facilitate review during debriefing. A time indication 
10 mark is a field of data containing the category and 

sub- category headings and a time code for the time that 
the mark is created. Preferably, a plurality of pre- 
defined time indication marks are organized by category 
and sub-category within a respective category (Blocks 
15 202 and 204) . An observer is presented with a computer 
display comprising a plurality of menus, windows, or 
the like, from which a particular time indication mark 
can be selected (Block 206) . A time code is associated 
with a selected time indication mark (Block 208) . 
20 For example, in an event including multiple 

interacting participants, if an observer witnesses 
participant behavior during the event that is 
particularly good or bad, a pre-defined time indication 
mark may be selected from the computer display and a 
25 time code associated with the selected mark. During 
debriefing, the event participants can view the 
datastreams in time synchronism beginning at or near 
the time represented by this time indication mark. 
Time indication marks that are not pre-defined may also 
30 be created and associated with the recorded 

datastreams. For example, if something were to occur 
during an event that would be worth reviewing during 
debriefing, yet does not fall within a particular pre- 
defined category, an observer is provided with the 
35 ability to create a time indication mark. 
Additionally, time indications marks may be 
automatically generated (Blocks 210 and 212) . For 
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example, certain events may trigger the creation of 
time indication marks, independent of an observer. 

Referring now to Fig. 1C, operations for 
retrieving and presenting the recorded datastreams in 
5 time synchronism via a marked indication of time will 
now be described. During debriefing of an event, a 
user is presented, preferably via a computer display, a 
list of time indication marks created during the event 
(Block 302) . If the user selects a time indication 

10 mark (Block 304) , the multiple recorded datastreams are 
displayed in time synchronism at or near the point in 
time represented by the time indication mark (Block 
306) . Preferably, time indication marks are displayed 
by category. However, a variety of display 

15 configurations may be used to facilitate the selection 
of a time indication mark by a user. 

The present invention permits various time 
indication marks to be associated with the multiple 
datastreams whereupon the datastreams can be accessed 

2 0 at these time indication marks during debriefing. The 

present invention also permits the time indication 
marks to be selected from a variety of categories, 
including technical aspects of the recorded event and 
human interaction aspects of the event participants. 
25 The invention is advantageous as a teaching tool for a 
variety of simulated events, especially those involving 
multiple participants, where risk and cost of the 
actual event are high. Exemplary events for which the 
debriefing system according to the present invention is 

3 0 advantageous include, but are not limited to, aircraft 

operations, land-based vehicle operations including 
buses and trains, surgical procedures, maritime 
simulations, and warfare simulations. The invention is 
also advantageous as a teaching tool for a variety of 
3 5 real events, especially those involving technical 
aspects of the event and multiple participants. 
Exemplary real event locations include, but are not 
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limited to, power plant control rooms, hospital 
operating rooms, space agency control centers,, war 
strategy rooms, operations centers of airports, air 
traffic control facilities, television broadcast rooms, 
5 the bridge of a ship, sports events, 

negotiation/mediation rooms, and various industrial and 
manufacturing facilities. 

In one embodiment, the present invention 
includes debriefing systems and methods used in 
10 conjunction with an aircraft flight simulator. Flight 
crew performance in a flight simulator exercise is 
generally measured via two components: technical 
proficiency and human factors or CRM proficiency. The 
technical component of flight crew performance 
15 includes, but is not limited to: knowledge and skills 
relating to aircraft systems; normal, abnormal and 
emergency operating procedures; air traffic control 
(ATC) procedures; instrument flight procedures; and 
navigation and charts. The CRM component of flight 
20 crew performance includes, but is not limited to: 

communication skills; team building; the exercise of 
command and use of leadership; decision making; and 
management of resources, workload and stress. 

The present invention comprises software, 
25 hardware, and interfaces necessary for capturing 

multiple datastreams from the flight simulator, and 
from video cameras and microphones in the simulator 
cockpit. Data from the flight simulator includes 
flight and engine instrumentation data, and other data 
3 0 required to create animated instrumentation and 
aircraft position displays. Typically, these 
datastreams are generated in digital format . On the 
other hand, the audio and video datastreams are 
typically generated in analog format. The present 
35 invention may convert analog datastreams into digital 
datastreams. Furthermore, the present invention can 
synchronize and store the multiple datastreams. An 
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important aspect of the present invention is the 
ability to create various time indication marks that 
can be associated with the recorded datastreams . The 
time indication marks permit virtually instant access 
5 to the multiple datastreams during debriefing . 

During a typical flight simulator debriefing 
session following a simulated flight exercise, the 
present invention displays the digitized video 
datastream in time synchronism with dynamic animation 

10 of flight instrumentation and plane position in a 
series of windows on a computer monitor. Audio 
playback, also in time synchronism with the displayed 
video and animated datastreams, is provided through a 
speaker system. Thus, plane position, instrument 

15 readings, and crew actions/interactions at any point in 
time can be viewed easily. During playback, the 
various synchronized datastreams can be fast forwarded, 
and reversed to virtually any point in time during the 
flight exercise with almost no delay. 

20 Referring now to Fig. 2, a preferred hardware 

' configuration of the present invention, when utilized 
as a debriefing system in conjunction with a flight 
simulator, is schematically illustrated. In general, 
the components of the debriefing system 10 include a 

25 flight simulator cab 20, a flight simulator host 
computer 30, a check pilot interface 40, a marking 
system 50, audio and video recording system 60, a 
debriefing computer 70, and a user interface 80 for the 
debriefing computer. As is known by those having skill 

30 in the art, a flight crew undergoes a flight training 
exercise within the flight simulator cab 20, which is 
controlled by the flight simulator host computer 30. 
The instrumentation within the flight simulator cab 20, 
which is typically a reproduction of the cockpit of a 

35 particular aircraft, is under computer control via the 
host computer 30. Control over a training exercise is 
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provided to a check pilot via the check pilot interface 
40. 

In the illustrated embodiment, the flight 
simulator cab 20 is fitted with a plurality of 
5 microphones 22 for recording the audible communications 
of flight crew members. In a typical configuration, 
four (4) cockpit microphones are used in conjunction 
with a cockpit -wide area microphone, Captain and First 
Officer boom microphones, and panel -mounted 
10 microphones. An audio mixer 23 combines the audio 

input from each microphone into a single audio source. 
The actions of the flight crew are recorded via one or 
more video cameras 24 located within the flight 
simulator cab 20. Preferably, each video camera 24 is 
15 an infrared low light camera. A video and audio 
booster 25 boosts the video and audio analog 
datastreams over long distances between the recording 
system 60 and the debriefing room computer 70, thereby 
facilitating the subsequent digitization of these 
20 datastreams. A hum buckler 26 is preferably provided 
to minimize audio and video datastream noise over long 
distances between the recording system 60 and the 
debriefing room computer 70. However, the present 
invention is not limited to analog video and audio 
25 recording devices. Digital recording devices, 

including charge coupled device (CCD) cameras, may be 
utilized. 

The present invention incorporates critical 
flight information with digitized audio and video 

3 0 obtained directly from the simulator. This information 
is instantly reproduced in a configurable display of 
the debriefing room computer 70. The monitor 
preferably provides a chase plane view from any vantage 
point which enhances situation awareness during 

35 debriefing. Preferably, the debriefing room computer 
70 is a Silicon Graphics Model R4000, or equivalent, 
having: audio and video capture and playback boards; 
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serial or Ethernet communications ports; a keyboard, a 
monitor suitable for viewing by small groups, a mouse, 
and data storage capacity of at least four (4) 
gigabytes. Even more preferable is data storage of at 
5 least eight (8) gigabytes. Software on the debriefing 
room computer 70 performs the following tasks: capture 
and store video and audio datastreams; store flight 
instrumentation datastreams and other datastreams from 
the simulator; store data from the marking system 

10 personal computer; replay audio and video datastreams; 
and replay flight simulator data in 4D animation. 4D 
animation provides the ability to view three 
dimensional (3D) animation from any point in space. 
Preferably, the movement of a computer mouse to any 

15 point around an object will allow a user to observe the 
object from that point in space. In addition, software 
is provided that makes the 4D animator speed up or slow 
down the frame rate replay to keep in synchronism with 
the video replay. 

20 Software is also provided that decodes 

simulator flight instrument indications into 
exceedances. The term exceedance refers to something 
that happened that should not have happened. For 
example, if an aircraft exceeds a maximum speed, this 

25 would be an exceedance. The maximum speed for an 
aircraft is stored in a computer file. The maximum 
speed may be further qualified by condition sets, such 
as maximum speed at a particular altitude. The program 
constantly compares the speed of the simulated aircraft 

30 with the file to determine if the maximum speed has 
been exceeded. If it has been exceeded, a mark and 
exceedance is created. 

Control over the debriefing computer 70 is 
maintained through a mouse -driven user interface, 

35 preferably having various menus for start up and 

configuration. In addition, it is preferred that the 
user interface have a panel similar to the protocol 
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used by a VCR interface, thereby facilitating ease of 
use. For example, "fast forward", "rewind", "play", 
"pause", and "stop" buttons are displayed on the 
screen. Preferably, buttons are provided for jumping 
5 to the previous or next time indication mark. The user 
can control playback during debriefing simply by 
clicking on the appropriate button displayed on the 
screen. All windows on the debriefing computer monitor 
are synchronized to remain in balance with the video 
10 during playback. Audio is provided through a speaker 
system linked with the debriefing computer and is fully 
synchronized with the video and graphics display. 

Referring now to Fig. 4, a preferred 
embodiment of the debriefing computer display 72, when 
15 used in conjunction with a flight simulator, is 

illustrated. Preferably, the display is a thirty- five 
inch (35") color monitor with at least four (4) 
different windows. A fifth window may be provided for 
displaying a list of time indication marks. The 
20 configurable display windows can be enlarged, moved, 
and reduced as desired. In the illustrated 
configuration, video display window 73 displays a 
digital video recording of a flight crew during a 
simulator exercise. Plane position display window 74 
25 illustrates the position of an airplane from a variety 
of view points at any stage during a simulated flight 
exercise. In addition, ideal plane positions may be 
superimposed over actual plane positions in the plane 
position display window 74. The plane position window 
30 74 also may display pertinent information such as glide 
slope and weather features. Instrumentation display 
windows 75 contain animated displays of selected 
aircraft instrumentation, configurable depending on the 
type of aircraft being "flown" in the simulator, and 
35 the particular simulated flight exercise involved. The 
animated instrumentation is a precise reproduction of 
the appearance of the actual instrumentation in the 
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cockpit of the simulator. The animation is dynamic in 
that the various instruments displayed appear to 
present data in real time. For example, the ground 
speed indicator and altimeter illustrated in Fig. 4 
5 give continuous readings during playback of a 
particular portion of the simulated exercise. 

All display windows are synchronized such 
that the dynamic instrumentation and plane position 
animation, and digital video and audio of flight crew 

10 actions, represent what transpired at a particular 
point in time during the recorded exercise. Because 
all displayed video and instrumentation data is in 
digital format, any point in time during the recorded 
flight exercise can be accessed instantly. In 

15 addition, as described in greater detail hereinafter, 
time indication marks can be created, both manually by 
the check pilot, automatically by an exceedance 
routine, and automatically by the simulator, at various 
points in time during the recording of the flight 

20 exercise so that, during debriefing, these particular 
points in time can be accessed instantly. 

Each time indication mark facilitates quick 
access to what transpired at that particular point in 
time during the simulated flight exercise when the 

25 recorded datastreams are played back during debriefing. 
During debriefing, the flight crew can observe their 
actions, listen to their communications, observe flight 
instrumentation and plane position prior to, 
simultaneously with, and subsequent to any time mark. 

30 For example, the recorded datastreams can be replayed 
beginning ten (10) seconds prior to a given time mark 
and continued therefrom. A plethora of replay options 
are made possible by the present invention, and are not 
limited to those described herein. The many possible 

35 replay options create a valuable tool for event 
participants to use in learning from an actual or 
simulated event. 
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The marking system 50 permits a check pilot 
to create time indication marks during a simulated 
flight exercise and to associate one or more categories 
of performance factors with the time marks. Preferably, 
5 the marking system 50 is controlled by a personal 
computer having a "touch screen" display, and is 
located in the flight simulator cab 20. The marking 
system user interface presented on the touch screen is 
described in detail below and is designed to facilitate 
10 ease of use by the check pilot. However, non-touch 
screen computer monitors may be utilized as well to 
present the marking system user interface . 
Conventional selection techniques including keyboard 
and mouse functions may also be used to create time 
15 indication marks. 

The marking system 50 includes software for 
producing a main menu and hierarchical sub-menus on the 
marking system display and for sending time mark data 
to the debriefing room computer 70. Preferably, the 
20 marking system 50 is connected to the debriefing room 
computer 70 via an Ethernet link or a serial port 
cable. In addition, the marking system 50 provides the 
check pilot with the ability to control the recording 
of the various video, audio, and instrumentation 
25 datastreams by the debriefing room computer 70. The 
marking system 50 instructs the debriefing system when 
to start, stop, and mark. 

During a flight simulation exercise, the 
check pilot is presented with a main menu 52, as shown 
30 at the top of Fig. 3, on the marking system display 
that includes the following categories of time 
indication marks: Human Factors, Technical, LOFT Event 
Trigger, Auto Detect, and End. From this main menu 52, 
the check pilot is presented with sub-categories 54 
35 within each category, as illustrated in Pig. 3. Each 
sub-category preferably contains a plurality of pre- 
defined time indication marks that can be selected as 
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described below. Pre-defined time indication marks and 
their respective categories and sub-categories may be 
presented to the user in a variety of ways, and are not 
limited to menus. Other methods of presenting 
5 categories and sub-categories of time indication marks 
to a user include using separate windows or icons. 

A time indication mark is defined as a field 
of data containing the main category heading, the sub- 
category heading(s), and a time code index at the time 

10 that the mark is received on the debriefing room 

computer 70. After each mark is selected, the time 
indication mark data is stored on the debriefing room 
computer 70, and the marking system 50 returns to the 
main menu of time mark categories, ready for the next 

15 time mark. 

Sub-categories of the Human Factors marks 
include, but are not limited to: "Briefings", 
"Inquiry/Assertion/Advocacy", "Crew Self Critique", 
"Communications/Decisions" , "Leadership/Followership" , 

20 "Interpersonal Relations", "Preparation/Planning/ 
Vigilance", and "Workload Management." These sub- 
categories are recommended by the Federal Aviation 
Administration (FAA) in the Crew Resource Management 
Advisory Circular (AC 120-51), which is incorporated 

25 herein by reference in its entirety. These sub- 
categories are CRM issues that airline training 
departments around the world have adopted as criteria, 
goals, and models towards which they strive to attain 
proficiency in. Tables 1-8 below set forth 

30 illustrative pre-defined time indication marks for each 
of the above -listed human factors sub - categories . 
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TABLE 1 



Briefings 



I — 

The effective briefing is interesting and thorough. It 
addresses coordination, planning, and problems. Although 
briefings are primarily a captain's responsibility, other crew 
members may add significantly to planning and should be 
encouraged to do so. 



10 



15 



20 



25 



30 



35 



40 



45 



Time Indication Marks and Descriptions 

The briefing establishes an environment 
for open/ interactive communications (for 
example, the captain calls for questions 
or comments, answers questions directly, 
listens with patience, does not interrupt 
or "talk over, " does not rush through the 
briefing, and makes eye contact as 
appropriate) . 

The briefing is interactive and emphasizes 
the importance of questions, critique, and 
the offering of information. 

The briefing establishes a "team concept" 
(for example, the captain uses "we" 
language, encourages all to participate 
and to help with the flight) . 

The brief ing covers pertinent safety and 
operational issues. 

The briefing identifies potential problems 
such as weather, delays, and abnormal 
system operations. 

The briefing provides guidelines for crew 
actions; division of labor and crew 
workload is addressed. 

The briefing includes the cabin crew as 
part of the team. 

The briefing sets expectations for 
handling deviations from standard 
operating procedures. 

The briefing establishes guidelines for 
the operation of automated systems (for 
example, when systems will be disabled; 
which programming actions must be 
verbalized and acknowledged) . 

The briefing specifies pilot flying and 
pilot not flying duties and 
responsibilities with regard to automated 
systems. 
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Inauirv/A dvocacv/Assertion 

These behaviors relate to crewmembers' promoting the course 
of action that they feel is best, even when it involves 
conflict with others. 



10 



15 



20 



Time Indication Marks and Descriptions 

Crewmembers speak up and state their 
information with appropriate persistence 
until there is some clear resolution. 

"Challenge and response 1 ' environment is 
developed. 

Questions are encouraged and are answered 
openly and nondef ensively . 

Crewmembers are encouraged to question the 
actions and decisions of others . 

Crewmembers seek help from others when 
necessary. 

Crewmembers question status and 
programming of automated systems to 
confirm situational awareness. 



25 



30 



35 



40 



Crew Self -Critique Regarding Decisions and Actions 

These behaviors relate to the effectiveness of a group 
and/or an individual crewmember in critique and debriefing. 
Areas covered should include the product , the process, and the 
people involved . Critique may occur during an activity, and/or 
after completing it. 

Time Indication Marks and Descriptions 

Critique occurs at appropriate times, 
which may be times of low or high 
workload . 

Critique deals with positive as well as 
negative aspects of crew performance. 

Critique involves the whole crew 
interactively. 

Critique makes a positive learning 
experience. Feedback is specific, 
objective, usable, and constructively 
given. 

Critique is accepted objectively and 

nondef ensively. _ 
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Communications /Decisions 

These behaviors relate to free and open communication. 
They reflect the extent to which crewmembers provide necessary 
information at the appropriate time (for example, initiating 
checklists and alerting others to developing problems) . Active 
participation in the decision making process is encouraged. 
Decisions are clearly communicated and acknowledged. 
Questioning of actions and decisions is considered routine. 

10 Time Indi cation Marks and Descriptions 

Operational decisions are clearly stated 
to other crewmembers. 

Crewmembers acknowledge their 
understanding of decisions. 

15 "Bottom lines" for safety are established 

and communicated. 

The "big picture" and the game plan are 
shared within the team, including flight 
attendants and others as appropriate. 

20 Crewmembers are encouraged to state their 

own ideas, opinions, and recommendations. 

Efforts are made to provide an atmosphere 
that invites open and free communications. 

Initial entries and changed entries to 
25 automated systems are verbalized and 

acknowledged. 
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Leadership Followerahip/Concern for Tasks 

These behaviors relate to appropriate leadership and 
followership. They reflect the extent to which the crew is 
concerned with the effective accomplishment of tasks. 

Time Indication Marks and Descriptions 

All available resources are used to 
accomplish the job at hand. 

Flight deck activities are coordinated to 
10 establish an acceptable balance between 

respect for authority and the appropriate 
practice of assertiveness . 

Actions are decisive when the situation 
requires . 

15 A desire to achieve the most effective 

operation possible is clearly 
demonstrated. 

The need to adhere to standard operating 
practices is recognized. 

20 Group climate appropriate to the 

operational situation is continually 
monitored and adjusted (for example, 
social conversation may occur during low 
workload, but not high) . 

25 Effects of stress and fatigue on 

performance are recognized. 

Time available for the task is well 
managed . 

Demands on resources posed by operation of 
30 | automated systems are recognized and 

managed . 

When programming demands could reduce 
situational awareness or create work 
overloads, levels of automation are 
35 reduced appropriately. 
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Interpersonal Rel ationships /Group Climat* 

These behaviors relate to the quality of interpersonal 
relationships and the pervasive climate of the flight deck. 

Time Indication Marks and Descriptions 

Crewmembers remain calm under stressful 
conditions . 

Crewmembers show sensitivity and ability 
to adapt to the personalities of others . 

10 Crewmembers recognize symptoms of 

psychological stress and fatigue in self 
and in others (for example, recognizes 
when he/she is experiencing "tunnel 
vision" and seeks help from the team; or 
15 notes when a crewmember is not 

communicating and draws him/her back into 
the team) . 

"Tone" in the cockpit is friendly, 
relaxed^ and supportive. 

2 0 During times of low communication, 

crewmembers check in with others to see 
how they are doing. 
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10 



15 



20 



25 



30 



35 



Preparation/Planning /Vigilance 

These behaviors relate to crews' anticipating contingencies 
and the various actions that may be required. Excellent crews 
are always "ahead of the curve" and generally seem relaxed. 
They devote appropriate attention to required tasks and respond 
without undue delay to new developments. (They may engage in 
casual social conversation during periods of low workload and 
not necessarily diminish their vigilance.) 

Time Indication Marks and Descriptions 

Demonstrating and expressing situational 
awareness; for example, the "model" of 
what is happening is shared within the 
crew. 

Active monitoring of all instruments and 
communications and sharing relevant 
information with the rest of the crew. 

Monitoring weather and traffic and sharing 
relevant information with the rest of the 
crew. 

Avoiding "tunnel vision" caused by stress; 
for example, stating or asking for the 
"big picture" . 

Being aware of factors such as stress that 
can degrade vigilance and watching for 
performance degradation in other 
crewmembers . 

Staying "ahead of the curve" in preparing 
for planned situations or contingencies. 

Ensuring that cockpit and cabin 
crewmembers are aware of plans . 

Including all appropriate crewmembers in 
the planning process. 

Allowing enough time before maneuvers for 
programming of the flight management 
computer . 

Ensuring that all crewmembers are aware of 
initial entries and changed entries in the 
flight management system. 
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10 



15 



20 



25 



Workload Distributed/Distractions Avoided 

These behaviors relate to time and workload management. 
They reflect how well the crew manages to prioritize tasks, 
share the workload, and avoid being distracted from essential 
activities . 

Time Indication Marks and Descriptions 

Crewmembers speak up when they recognize 
work overloads in themselves or in others. 

Tasks are distributed in ways that 
maximize efficiency. 

Workload distribution is clearly 
communicated and acknowledged. 

Nonoperational factors such as social 
interaction are not allowed to interfere 
with duties. 

Task priorities are clearly communicated. 

Secondary operational tasks (for example, 
dealing with passenger needs and 
communications with company) are 
prioritized so as to allow sufficient 
resources for primary flight duties. 

Potential distractions posed by automated 
systems are anticipated, and appropriate 
preventive action is taken, including 
reducing or disengaging automated features 
as appropriate. 



Should the check pilot in the simulator 
observe either positive or negative behavior in these 
30 sub- categories, he or she will indicate that 

observation by creating one of the above, or other, 
time indication marks. Preferably, the creation of the 
time indication mark is accomplished by touching one of 
these listed time indication marks on the touch screen. 

3 5 However, a time indication mark may be created in ways 

other than via a touch screen, such as via a mouse or 
keyboard function. In addition, time indication marks 
may be created that are not pre-defined within a 
particular category or sub-category. For example, if 

4 0 something were to occur during a simulated event that 

would be worth reviewing during debriefing, yet does 
not fall within a particular pre-defined category, a 
check pilot can create a generic time indication mark 
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to mark this particular point in time during the 
simulated event . 

Technical sub-categories of time indication 
marks include the minimum number of maneuvers that a 
5 flight crew should be able to perform proficiently. 
These sub- categories may vary from one country to 
another, but the majority of them are standard among 
airlines of the world. Generally, these sub- categories 
include, but are not limited to: "Rejected Takeoff", 

10 "VI Cut", "Area Departure", "Area Arrival", Non 

Precision Approaches", "Holding", "ILS Approaches", 
"Missed Approaches", "Landings", and "Windshear." As 
shown, various sub-categories of these sub-categories 
exist. The check pilot in the simulator may select 

15 time indication marks from any of the technical sub- 
categories . 

"LOFT" is the acronym for "Line Oriented 
Flight Training." The airline industry has been moving 
away from proficiency-based maneuvers of a technical 

20 nature and more towards simulations that are closely 
oriented towards actual line operations. The term 
"line operations" refers to flying with passengers. 
The emphasis is on human factors training and 
evaluation, rather than on motor skills, training, 

25 demonstration, and evaluation. A "LOFT" event trigger 
is defined as an external event that is introduced into 
a simulated flight exercise and that is specifically 
designed to induce a human factors event. For example, 
an engine oil temperature gauge may show an erratic 

30 reading during a simulated flight to distract the 

flight crew. The distraction may lead to failure of 
workload management or communications between the 
flight crew. The focus during debriefing would be to 
manage the distraction so that the flight may 

35 successfully continue notwithstanding the distraction. 
The check pilot in the simulator may select time 
indication marks from any LOFT categories and sub- 
categories . 
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"Auto Detect" includes various automatically- 
created time indication mark sub-categories. Exemplary 
sub- categories include: when the check pilot initiates 
certain simulator functions, and when certain 
5 performance parameters are exceeded. For example, a 
time indication mark may be automatically created when 
the check pilot requests the simulator to induce 
windshear. As another example, a time indication mark 
may be automatically created when the aircraft is 
10 operated in excess of a particular airspeed at a 
particular altitude. Automatically created time 
indication marks can be numerous and it is expected 
that users will customize the present invention to fit 
particular needs. 
15 Referring now to Fig. 5, a debriefing room 

computer display screen 82 for selecting a particular 
time indication mark from a list of time indication 
marks 83 during debriefing is illustrated. A user 
enters the number of the time indication mark in the 
20 field labeled "Enter Mark #" and presses a keyboard or 
mouse function to begin playback of the various 
datastreams at the point in time indicated by the 
selected mark. The screen 82 permits the user to view 
additional screen listings of time marks through "next 
25 page" and "previous page" functions and their 

equivalents. Alternatively, a display of elapsed time 
at a particular mark may be presented along with 
controls for playback from a selected elapsed time. 

As shown, a time code 84, representing a 
30 point in time during a particular recorded flight 

exercise, is associated with each time indication mark. 
In addition, both a mark category 85 and sub-category 
86 are associated with each time indication mark. For 
example, as shown in Fig. 5, at a time equal to ten 
35 (10) seconds into a particular flight exercise, a time 
mark was created for a "VI Cut" technical maneuver. 
Upon selecting this mark during debriefing, the display 
illustrated in Fig. 4 would replay, in digital format 
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and in time synchronism, the video of the flight crew, 
the plane position, and various flight instrumentation 
beginning at ten (10) seconds into the flight exercise. 
However, it is to be understood that the present 
5 invention permits replay to be configurable whereupon 
it can begin and end before, after, or precisely at the 
ten (10) second mark. 

Referring now to Fig. 6, another debriefing 
room computer display screen 88 for selecting time 

10 indication marks during debriefing is illustrated. The 
illustrated screen displays a plot 91 of aircraft 
altitude during a particular segment of a flight 
exercise. Altitude in feet is plotted along the "y" 
axis 89, and time is plotted along the "x" axis 90. 

15 Vertical marks 92 represent time indication marks 

created via the marking system 50 during a simulated 
flight exercise. During the simulated flight exercise, 
the marks were either automatically created by the 
simulator, or by the check pilot as described 

20 previously. 

During debriefing, a user selects a 
particular mark by "clicking" on the mark with a mouse. 
When selected, the details of what happened at that 
point in time can be displayed on the debriefing room 

25 computer monitor as previously described above. In 

addition, when a mark is selected, preferably a window 
containing the data for the selected time indication 
mark is displayed. Preferably, the time marks are 
color-coded based on the type of mark to facilitate 

30 easy recognition. For example, some or all of the 

vertical marks 92 in Fig. 6 may be magenta to represent 
Technical marks. Some or all of the vertical marks 92 
in Fig. 6 may be red to represent Human Factors marks. 
Some or all of the vertical marks 92 in Fig. 6 may be 

35 yellow may to represent LOFT Event Trigger marks. Some 
or all of the vertical marks 92 in Fig. 6 may be white 
to represent Auto Detect marks. In addition, the 
screen has zoom- in and zoom-out capability, to 
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facilitate easy use should an area of the screen become 
cluttered with marks. 

Referring now to Fig. 7, the main menu 52 on 
the marking system display may include a category for 
5 voice recognition. Audio datastreams produced by 
recording the audible communications of flight crew 
members may be analyzed using voice recognition 
software. As is known to those having skill in the 
art, words, phrases and other portions of an audio 
10 datastream may be selectively located using the voice 
recognition software. Preferably, keyword searching is 
provided to facilitate locating selective portions of 
an audio datastream. For example, a search could be 
performed during debriefing to determine whether the 
15 crew used the word "abort" or "fire" during the 

simulated exercise. Selectively located words and 
other portions of an audio datastream may serve as time 
indication marks and may be stored for later retrieval. 

According to another aspect of the present 
20 invention, video examples of appropriate flight crew 
behavior, may be instantly available for a particular 
event during a simulator exercise. For example, if a 
CRM error occurs during a session, the system, if 
prompted during debriefing or briefing will respond 
25 with a video example of crew actions/interactions that 
would be appropriate for the particular event. 

The systems and methods according to the 
present invention can be used as a teaching tool for a 
variety of events. For example, during a surgical 
30 procedure, a video camera can be used to record the 

event. The system can be configured to interface with 
the video camera as well as with instrumentation used 
during the procedure, such as monitors, respirators, 
and the like. The video and/or audio signal is 
35 digitized and synchronized with the instrumentation 

data, and then stored. Subsequently, the stored video 
and instrumentation data can be displayed, thereby 
showing the actions/interactions of a surgical team 
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simultaneously with various instrumentation readings. 
The end result is a valuable teaching aid for use in 
debriefing the surgical team and for training others. 

The present invention is a vast improvement 
5 over existing tools for debriefing training exercises 
and real events. The present invention increases 
training efficiency and effectiveness by allowing 
trainees to M re-live" their training or operations 
experience via a single integrated display of their 

10 performance with respect to both technical aspects and 
human factors aspects. 

The foregoing is illustrative of the present 
invention and is not to be construed as limiting 
thereof. Although a few exemplary embodiments of this 

15 invention have been described, those skilled in the art 
will readily appreciate that many modifications are 
possible in the exemplary embodiments without 
materially departing from the novel teachings and 
advantages of this invention. Accordingly, all such 

20 modifications are intended to be included within the 
scope of this invention as defined in the claims. In 
the claims, means-plus-function clauses are intended to 
cover the structures described herein as performing the 
recited function and not only structural equivalents 

25 but also equivalent structures. Therefore, it is to be 
understood that the foregoing is illustrative of the 
present invention and is not to be construed as limited 
to the specific embodiments disclosed, and that 
modifications to the disclosed embodiments, as well as 

30 other embodiments, are intended to be included within 
the scope of the appended claims. The invention is 
defined by the following claims, with equivalents of 
the claims to be included therein. 
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THAT WHICH IS CLAIMED: 

1. An event debriefing system comprising: 
means for recording a plurality of 

datastreams related to an event which occurs over an 
5 extended time period, each datastream including 
chronological information related to at least one 
aspect of the event over the extended time period; 

categorized marking means, for marking times 
during the extended time period and for associating a 
10 category with the marked times; and 

presenting means, responsive to said marking 
means, for retrieving information from said datastreams 
based upon the marked times in a selected category, and 
for presenting the retrieved information from the 
15 datastream for the marked times in the selected 
category. 

2. An event debriefing system according to 
Claim 1, wherein said associated category includes a 
category related to actions of human participants in 

20 said event. 

3. An event debriefing system according to 
Claim 1, wherein said associated category includes a 
category related to technical parameters of said event. 

4. An event debriefing system according to 
claim 1, wherein said synchronized presenting means 
comprises means for retrieving, information from said 
datastreams beginning at the marked time and continuing 
chronologically thereafter, and for presenting the 
retrieved information from the datastreams in time 
synchronism. 



25 
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5. An event debriefing system according to 
Claim 1, wherein said recording means comprises: 

video recording means for producing a video 
datastream of actions of event participants; 
5 audio recording means for producing an audio 

datastream of audible communications of event 
participants; and 

instrumentation recording for producing an 
instrumentation datastream from various event 
10 monitoring instruments. 



6. An event debriefing system according to 
Claim 5 wherein said synchronized presenting means 
comprises a display having a plurality of zones, and 
wherein said video and instrumentation datastreams are 

15 presented in a respective one of said plurality of 
zones in time synchronism. 

7. An event debriefing system according to 
Claim 6 wherein said synchronized presenting means 
comprises means for performing said audio datastream 

20 with said displayed video and instrumentation 
datastreams in time synchronism. 

8. An event debriefing system according to 
Claim 7 wherein said performing means further comprises 
voice recognition means for selectively locating 

25 portions of said audio datastream. 

9. An event debriefing system according to 
Claim 1 wherein said marking means is responsive to 
user identification of a time during said extended time 
period. 
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10. An event debriefing system according to 
Claim 1, wherein said synchronized presenting means 
includes means for displaying the appearance of 
selected instrumentation. 

5 11. An event debriefing system comprising: 

means for recording a plurality of 
datastreams related to an event which occurs over an 
extended time period, each datastream including 
chronological information related to at least one 
10 aspect of the event over the extended time period; 

automatic marking means, for marking a time 
during the extended time period wherein at least one of 
said aspects meets predetermined criteria; and 
presenting means, responsive to said 
15 automatic marking means, for retrieving information 
from said datastreams based upon the automatically 
marked time, and for presenting the retrieved 
information from the datastreams. 

12. An event debriefing system according to 
claim 11 wherein said synchronized presenting means 
comprises means for retrieving information from said 
datastreams beginning at the marked time and continuing 
chronologically thereafter, and for presenting the 
retrieved information from the datastreams in time 
synchronism. 

13 . An event debriefing system according to 
Claim 11 wherein said recording means comprises : 

video recording means for producing a video 
datastream of actions of event participants; 
30 audio recording means for producing an audio 

datastream of audible communications of event 
participants; and 
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instrumentation recording for producing an 
instrumentation datastream from various event 
monitoring instruments. 

14. An event debriefing system according to 
5 Claim 13 wherein said synchronized presenting means 
comprises a display having a plurality of zones, and 
wherein said video and instrumentation datastreams are 
presented in a respective one of said plurality of 
zones in time synchronism. 

10 15. An event debriefing system according to 

Claim 14 wherein said synchronized presenting means 
comprises means for performing said audio datastream 
with said displayed video and instrumentation 
datastreams in time synchronism. 

15 16. An event debriefing system according to 

Claim 11 further comprising categorized marking means 
for marking time during the extended tim period and for 
associating a category with the marked times. 

17. An event debriefing system according to 
20 Claim 11, wherein said synchronized presenting means 

includes means for displaying the appearance of 
selected instrumentation. 

18. An aircraft flight simulator debriefing 

system, comprising: 

an aircraft flight simulator for simulating 

aircraft flight; 

means for recording a plurality of 
datastreams related to a simulated aircraft flight, 
each datastream including chronological information 
related to at least one aspect of the simulated 
aircraft flight; 
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categorized marking means, for marking times 
during the simulated aircraft flight and for 
associating a category with the marked times; and 

presenting means, responsive to said marking 
5 means, for retrieving information from said datast reams 
based upon the marked times in a selected category, and 
for presenting the retrieved information from the 
datastream for the marked times in the selected 
category. 

10 19. An aircraft flight simulator debriefing 

system according to Claim 18, wherein said associated 
category includes a category related to actions of 
human participants in said simulated aircraft flight. 

20. An aircraft flight simulator debriefing 
15 system according to Claim 18, wherein said associated 

category includes a category related to technical 
parameters of said simulated aircraft flight. 

21. An aircraft flight simulator debriefing 
system according to claim 18, wherein said synchronized 

2 0 presenting means comprises means for retrieving 

information from said datastreams beginning at the 
marked time and continuing chronologically thereafter, 
and for presenting the retrieved information from the 
datastreams in time synchronism. 

22. An aircraft flight simulator debriefing 
system according to Claim 18, wherein said recording 
means comprises: 

video recording means for producing a video 
datastream of actions of event participants; 

audio recording means for producing an audio 
datastream of audible communications of event 
participants; and 
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instrumentation recording for producing an 
instrumentation datastream from various event 
monitoring instruments. 

23. An aircraft flight simulator debriefing 
5 system according to Claim 22 wherein said synchronized 
presenting means comprises a display having a plurality 
of zones, and wherein said video and instrumentation 
datastreams are presented in a respective one of said 
plurality of zones in time synchronism. 

10 24 . An aircraft flight simulator debriefing 

system according to Claim 23 wherein said synchronized 
presenting means comprises means for performing said 
audio datastream with said displayed video and 
instrumentation datastreams in time synchronism. 



15 25. An aircraft flight simulator debriefing 

system according to Claim 24 wherein said performing 
means further comprises voice recognition means for 
selectively locating portions of said audio datastream. 

26. An aircraft flight simulator debriefing 
2 0 system according to Claim 18 wherein said marking means 

is responsive to user identification of a time during 
said simulated aircraft flight. 

27. An aircraft flight simulator debriefing 
system according to Claim 18, wherein said synchronized 

25 presenting means includes means for displaying the 

appearance of selected aircraft flight instrumentation. 

28. A method of event debriefing comprising 

the steps of : 

recording a plurality of datastreams related 
30 to an event which occurs over an extended time period, 
each datastream including chronological information 
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related to at least one aspect of the event over the 
extended time period; 

automatically marking a time during the 
extended time period wherein at least one of said 
5 aspects meets predetermined criteria; and 

retrieving information from said datastreams 
based upon the automatically marked time, and 
presenting the retrieved information from the 
datast reams in time synchronism. 

29. A method of event debriefing according 
to Claim 28 wherein said step of retrieving information 
from said datastreams based upon the marked time 
comprises retrieving information from said datastreams 
beginning at the marked time and continuing 
chronologically thereafter, and presenting the 
retrieved information from the datastreams in time 
synchronism. 

30. A method of event debriefing according 
to Claim 28 wherein said step of recording a plurality 
of datastreams related to an event comprises: 

producing a video datastream of actions of 
event participants; 

producing an audio datastream of audible 
communications of event participants; and 

producing an instrumentation datastream from 
various event monitoring instruments. 

31. A method of event debriefing, according 
to Claim 3 0 wherein said step of retrieving and 
presenting the retrieved information comprises 

3 0 displaying said video and instrumentation datastreams 
in a respective one of a plurality of display zones in 
time synchronism. 
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32. A method of event debriefing according 
to Claim 31, further comprising the step of performing 
said audio datastream with said displayed video and 
instrumentation datastreams in time synchronism. 

5 33. A method of event debriefing according 

to Claim 32 wherein said performing step further 
comprises the step of selectively locating voice 
portions of said audio datastream. 

34. A method of event debriefing according 
to Claim 28, wherein said step of automatically marking 
a time during the extended time period further 
comprises : 

associating a category with the marked times; 

and 

retrieving information from said datastreams 
based upon the marked times in a selected category, and 
presenting the retrieved information from the 
datastream for the marked times in the selected 
category. 

20 35. A method of event debriefing according 

to Claim 34, wherein said step of associating a 
category includes associating a category related to 
actions of human participants in said event with the 
marked times. 

25 36. A method of event debriefing according 

to Claim 34, wherein said step of associating a 
category includes associating a category related to 
technical parameters of said event . 
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recording a plurality of datastreams related 
to a simulated aircraft flight, each datastream 
including chronological information related to at least 
one aspect of the simulated aircraft flight; 
5 marking times during the simulated aircraft 

flight and for associating a category with the marked 
times; and 

retrieving information from said datastreams 
based upon the marked times in a selected category, and 
10 for presenting the retrieved information from the 
datastream for the marked times in the selected 
category, 

38. A method of debriefing an aircraft 
flight simulator exercise according to Claim 37, 

15 wherein said step of retrieving information from said 
datastreams based upon the marked time comprises 
retrieving information from said datastreams beginning 
at the marked time and continuing chronologically 
thereafter, and presenting the retrieved information 

20 from the datastreams in time synchronism. 

39. A method of debriefing an aircraft 
flight simulator exercise according to Claim 37, 
wherein said step of recording a plurality of 
datastreams related to an event comprises: 

25 producing a video datastream of actions of 

flight simulator exercise participants; 

producing an audio datastream of audible 

communications of flight simulator exercise 

participants; and 
30 producing an instrumentation datastream from 

various flight simulator flight exercise monitoring 

instruments. 
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40. A method of debriefing an aircraft 
flight simulator exercise according to Claim 3 9 wherein 
said step of retrieving and presenting the retrieved 
information comprises displaying said video and 

5 instrumentation datastreams in a respective one of a 
plurality of display zones in time synchronism. 

41. A method of debriefing an aircraft 
flight simulator exercise according to Claim 40, 
further comprising the step of performing said audio 

10 datastream with said displayed video and 

instrumentation datastreams in time synchronism. 

42. A method of debriefing an aircraft 
flight simulator exercise according to Claim 41 wherein 
said performing step comprises the step of selectively 

15 locating voice portions of said audio datastream. 

43. A method of debriefing an aircraft 
flight simulator exercise according to Claim 37, 
wherein said step of marking a time during the extended 
time period further comprises: 

20 associating a category with the marked times; 

and 

retrieving information from said datastreams 
based upon the marked times in a selected category, and 
presenting the retrieved information from the 
25 datastream for the marked times in the selected 
category. 

44. A method of debriefing an aircraft 
flight simulator exercise according to Claim 43, 
wherein said step of associating a category includes 

3 0 associating a category related to actions of human 

participants in said flight simulator exercise with the 
marked times. 
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45. A method of debriefing an aircraft 
flight simulator exercise according to Claim 43, 
wherein said step of associating a category includes 
associating a category related to technical parameters 

5 of said flight simulator exercise. 

46 . A method of debriefing an aircraft 
flight simulator exercise according to Claim 3 7 further 
comprising : 

marking a time during the extended time 
10 period wherein at least one of said aspects meets 
predetermined criteria; and 

retrieving information from said datastreams 
based upon the automatically marked time, and for 
presenting the retrieved information from the 
15 datastreams. 
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